<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Communication &mdash; SIMULATeQCD 0.7 documentation</title>
      <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
      <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
      <link rel="stylesheet" href="../_static/togglebutton.css" type="text/css" />
      <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
        <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/togglebutton.js"></script>
        <script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
        <script async="async" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
        <script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
    <script src="../_static/js/theme.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="Functor syntax" href="functorSyntax.html" />
    <link rel="prev" title="Lattice Container" href="latticeContainer.html" /> 
</head>

<body class="wy-body-for-nav"> 
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #343131" >
            <a href="../index.html" class="icon icon-home"> SIMULATeQCD
          </a>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>
        </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../01_gettingStarted/gettingStarted.html">Getting started</a></li>
<li class="toctree-l1"><a class="reference internal" href="../02_contributions/contributions.html">Contributions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../03_applications/applications.html">Applications</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="codeBase.html">Code base</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="indexer.html">Indexer</a></li>
<li class="toctree-l2"><a class="reference internal" href="fileWriter.html">File Writer</a></li>
<li class="toctree-l2"><a class="reference internal" href="latticeContainer.html">Lattice Container</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Communication</a></li>
<li class="toctree-l2"><a class="reference internal" href="functorSyntax.html">Functor syntax</a></li>
<li class="toctree-l2"><a class="reference internal" href="gaugefield.html">Gaugefield</a></li>
<li class="toctree-l2"><a class="reference internal" href="topology.html">Topology</a></li>
<li class="toctree-l2"><a class="reference internal" href="configurationIO.html">Input and output formats for gauge configurations</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../05_modules/modules.html">Modules</a></li>
</ul>

        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu"  style="background: #343131" >
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">SIMULATeQCD</a>
      </nav>

      <div class="wy-nav-content">
        <div class="rst-content">
          <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
          <li><a href="codeBase.html">Code base</a> &raquo;</li>
      <li>Communication</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../_sources/04_codeBase/communicationBase.md.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
             
  <section class="tex2jax_ignore mathjax_ignore" id="communication">
<h1>Communication<a class="headerlink" href="#communication" title="Permalink to this headline"></a></h1>
<p>All classes related to communication can be found in</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>src/base/communication
</pre></div>
</div>
<p>To work with multiple devices, SIMULATeQCD splits a lattice into multiple
sublattices, with partitioning possible along any
space-time direction. Each sublattice is given to a single GPU, and we
call this sublattice the bulk.
In addition, the GPU holds a copy of the outermost borders of neighboring
sublattices, which we call the halo. This halo is necessary because many
measurement and update processes are stencil operations, which means that a
calculation performed at some site may need information from a neighboring
sublattice. Every so often, information from all sublattices must be injected
into their neighbors’ halos. A schematic drawing of the exchange
of halo information between different GPUs is shown below.</p>
<p><img alt="alt" src="../_images/haloStructure.png" /></p>
<section id="communicationbase">
<h2>CommunicationBase<a class="headerlink" href="#communicationbase" title="Permalink to this headline"></a></h2>
<p>Communication between multiple CPUs and multiple nodes is handled with MPI, which also
allows for communication between multiple GPUs.
We use MPI two-sided communication.
For NVIDIA hardware, we handle communication between GPUs on the same node using CUDA
GPUDirect P2P. CUDA-aware MPI is used for internode communication.
We boost performance by allowing the code to
carry out certain computations while communicating, such as copying halo buffers
into the bulk, whenever possible.
An example of what different
communication channels might be available for two nodes is given below.</p>
<p>Wrappers for methods used in these various communication libraries are collected in
the <code class="docutils literal notranslate"><span class="pre">CommunicationBase</span></code> class. The <code class="docutils literal notranslate"><span class="pre">CommunicationBase</span></code> will also detect whether
CUDA-aware MPI or GPUDirect P2P are available, and if they are, use them
automatically since these channels have less communication overhead than standard MPI,
and are hence much faster.</p>
<p><img alt="alt" src="../_images/communicationChannel.png" /></p>
</section>
<section id="halooffsetinfo-and-neighborinfo">
<h2>HaloOffsetInfo and neighborInfo<a class="headerlink" href="#halooffsetinfo-and-neighborinfo" title="Permalink to this headline"></a></h2>
<p>Halo communication proceeds by first copying halo information contiguously into a buffer.
This requires translating from the sublattice’s indexing scheme to the buffer’s indexing
scheme; the difference between these schemes is called the halo offset,
computed by <code class="docutils literal notranslate"><span class="pre">haloOffsetInfo</span></code> class.
This class provides offsets for different halo segments
(stripe halo, corner halo, etc). These offsets and the buffer
base pointer are used to place the halo data at the correct position in the buffer.
An example for the corner halo would be:</p>
<p><span class="math notranslate nohighlight">\(
\text{corner buffer pointer} = \text{buffer base pointer} +
                                   \text{corner halo offset}.
\)</span></p>
<p>In order to communicate, each sublattice needs to know something about his neighbors,
for example their rank or whether they are on the same node. This information is
kept in the <code class="docutils literal notranslate"><span class="pre">neighborInfo</span></code> class.</p>
</section>
<section id="sitecomm">
<h2>SiteComm<a class="headerlink" href="#sitecomm" title="Permalink to this headline"></a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">SiteComm</span></code> class is the highest level class from which all objects that need to
communicate across sublattices, such as the <a class="reference internal" href="gaugefield.html#gaugefield"><span class="std std-doc">Gaugefield</span></a>
inherit. It uses the <a class="reference internal" href="../02_contributions/memoryAllocation.html"><span class="doc std std-doc">Memorymanagement</span></a>
to allocate memory for the buffer,
uses the <code class="docutils literal notranslate"><span class="pre">haloOffsetInfo</span></code> to translate the local index to the buffer index,
copies information into the buffer, and finally uses the <code class="docutils literal notranslate"><span class="pre">CommunicateBase</span></code> to
carry out the exchange.</p>
</section>
</section>


           </div>
          </div>
          <footer>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2021, LatticeQCD.</p>
  </div>

  Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    provided by <a href="https://readthedocs.org">Read the Docs</a>.
   

</footer>
        </div>
      </div>
    </section>
  </div>
  <script>
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script> 

</body>
</html>